Image processing device and image display apparatus

ABSTRACT

An image processing device includes a background luminance input unit, an image processing unit and a luminance setting unit. The background luminance input unit stores background luminance information regarding background luminance. The image processing unit generates display image data and contrast conversion information. The display image data is provided by modulating a contrast of a correction image data so that pixel values included in the correction image data are in an acceptable range. The correction image data is provided by correcting blurring in an input image data to be input to the image processing unit based on the background luminance information. The contrast conversion information is information used for modulating the contrast. The luminance setting unit generates a luminance setting information to set a luminance of the projection unit based on the modulation information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2015-076102, filed Apr. 2, 2015, theentire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to an image processingdevice and image display apparatus.

BACKGROUND

Image display apparatus for displaying images to the eyes of a user havebeen proposed. For example, the image display apparatus may be ahead-mounted display. The image display apparatus may comprise a displayunit, a projection unit, and a reflection unit. Light from a lightsource incident to the display unit and light including imageinformation based on display image data is emitted by the display unit.The direction of at least a part of the light rays of the lightincluding image information is corrected by an optical component, andthe projection unit emits the corrected light. The reflection unitreflects at least a part of the light rays of the corrected light towardan eye of a user.

An image (an observation image) observed by a user may include blurring,color shift, and/or distortion due to an aberration caused by opticalcomponents or lens systems in the reflection unit, such as lenses orhalf mirrors. Image processing techniques to suppress this deteriorationin image quality have been proposed. An inverse of the deteriorationcharacteristics due to the aberration is applied to the image data to beinput to the image display apparatus prior to display. In particular, aninverse filter having inverse characteristics relative to a space filteror frequency filter that expresses the blurring aberration in theobservation image may be applied to the display image data. The user isthus able to observe an image in which the blurring is suppressed by theinverse filter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an image display apparatus according to a firstembodiment.

FIG. 2 is a block diagram of the image display apparatus according tothe first embodiment.

FIG. 3 is a block diagram of a deblurring unit according to the firstembodiment.

FIG. 4 is a flow chart corresponding to the FIG. 2.

FIG. 5 is a flow chart corresponding to the FIG. 3.

FIG. 6 is a block diagram of the image display apparatus according to asecond embodiment.

FIG. 7 is a flow chart corresponding to the FIG. 6.

FIG. 8 is a diagram of a structure of an image display apparatus.

DETAILED DESCRIPTION

Each of the embodiments will now be described in detail with referenceto the accompanying drawings.

Note that the figures are conceptual pattern diagrams, and therelationships between thicknesses and widths and ratios of size of eachpart are not necessarily represented to scale. Moreover, the size andratio of components that appear in multiple figures are not necessarilythe same in each figure.

According to one embodiment, an image processing device includes abackground luminance input unit, an image processing unit and aluminance setting unit. The background luminance input unit storesbackground luminance information regarding background luminance. Theimage processing unit generates display image data and contrastconversion information. The display image data is provided by modulatinga contrast of a correction image data set so that pixel values includedin the correction image data set are in an acceptable range. Thecorrection image data set is provided by correcting blurring in an inputimage data set to be input to the image processing unit based on thebackground luminance information. The contrast conversion information isinformation used for modulating the contrast. The luminance setting unitgenerates luminance setting information to set a luminance of theprojection unit based on the modulation information.

(First Embodiment)

FIG. 1 is a diagram of an image display apparatus according to a firstembodiment. As illustrated in FIG. 1, an image display apparatus 100comprises a circuit (image processing device) 110, a projection unit120, and a reflection unit 130.

The circuit 110 is connected to an external memory medium or network,and receives input image data. The connection may be wired or wireless.The circuit 110 may be connected to the projection unit electricallythrough a cable 115 and may transmit the image display data to a display122 in the projection unit 120.

The projection unit 120 emits light including image information based ondisplay setting information.

The projection unit 120 may comprise a light source 121, the display122, and a lens system 123. The light source 121 may be white LED (LightEmitting Diode) and may comprise a red LED, a green LED, and a blue LED.

The display 122 may comprise a plurality of pixels arranged in a planeto display an image based on the display image data. The display 122transmits light L1 emitted from the light source 121. Light includingimage information L2 based on the display image data is emitted by thedisplay 122 toward the lens system 123. The light source 121 may notnecessarily be needed in the case where the display 122 is aself-lighting type display. The display 122 may, for example, comprise aliquid crystal display, an organic EL (Electro Luminescence) display orLCOS (Liquid Crystal On Silicon) display.

The lens system 123 is located between the display and the reflectionunit 130 in a light path of the light including image information L2.The lens system 123 corrects a direction of at least a part of the lightincluding image information to travel toward the reflection unit 130.The light that travels from the lens system 123 toward the reflectionunit 130 is defined as correction light L3.

The lens system 123 includes at least one optical component. The opticalcomponent may be, for example, a lens, a prism, or a mirror. In the casewhere the lens system 123 includes a plurality of optical components,the optical components are not necessarily directly-aligned.

The reflection unit 130 reflects at least a part of light emitted by theprojection unit toward the eye 161 of a viewer. That is, the reflectionunit 130 reflects at least a part of light rays included in thecorrection light L3 towards the eye 161, so the reflection unit 130reflects at least a part of the light including image information towardthe eye. The reflection unit 130 transmits a part of incident light sothat a user observes background on which an image overlaps. At least apart of the light rays included in the correction light L3 reflected bythe reflection unit 130 is incident on the pupil 162 and is collected onthe retina 163 by the lens and cornea. A virtual image is formed on anextended line of light rays included in the correction light L3 extendedin a direction opposite to the traveling direction. In this way, it ispossible to view the observation image 170 for a user 160.

In this embodiment, the relative position between the display 122 and alens system 123 is fixed in the projection unit 120. The relativeposition between the display 122 and an lens system 123 may not be fixedin the projection unit 120. For example, the distance and relativeattitude between the display 122 and the lens system 123 may be adjustedby adjusting screws that fix the position of the display 122 and/or thelens system 123 on the projection unit. For example, the distancebetween the pupils 162 and the observation image 170 and the size of theobservation image 170 may be adjusted by adjusting the distance betweenthe display 122 and the lens system 123.

The reflection unit 130 transmits a part of a light that is incident onthe reflection unit 130. In this manner, a user is able to see the viewthrough the reflection unit 130.

The reflection unit 130 is provided along a first plane 131. Forexample, the reflection unit 130 may comprise a plurality of finereflection planes arranged along the first plane 131. The first plane131 may be flat or curved. Each of the plurality of reflection planesmay be implemented using a half mirror that reflects at least a part ofthe light that is incident on the reflection plane. However, thereflection unit 130 need not be implemented using a half mirror. Anymember that transmits a part of the incident light and reflects orotherwise modifies a reflection angle of another part of the incidentlight may be used as the reflection unit 130.

Each of the plurality of reflection planes is declined with respect tothe first plane 131 and there are gaps between the reflection planesalong a direction perpendicular to the first plane 131. The anglebetween the reflection plane and the first plane 131 is set based on thelight axis of the projection unit 120 and the positional relationshipbetween the eyes 161 and the observation image 170. Reflection angle ischanged by the angle between the reflection plane and the first plane131. The reflection unit 130 may be implemented by a plurality ofreflection planes, e.g., forming a Fresnel mirror.

The observation image 170 is displayed in front of a user 160 in thisembodiment. The observation image 171 may alternatively be displayed atan edge of the range of the user's vision, so as not to block the user'sview.

The image display apparatus 100 may be implemented in a heads-updisplay. The image display apparatus 100 may be, for example, aspectacle type image display apparatus. The image display apparatus 100may include a spectacle frame or holding part 140. FIG. 1 shows imagedisplay apparatus 100 mounted on the head of a user 160 with the holdingpart 140.

The image display apparatus 100 may further include spectacle lenses150. The reflection units 130 may be included in the spectacle lenses150. The holding part 140 may include a nose pad 141, bridge 142, andtemple 143. The bridge 142 portion may connect spectacle lenses 150. Theholding part 140 may further include a rim to hold the spectacle lenses150. A non-refractive glass may be used as the spectacle lenses 150.Each of the spectacle lenses 150 has a first surface 151 and a secondsurface 152 being apart from the first surface 151. The reflection units130 may be provided between the first surface 151 and the second surface152. The reflection units 130 may additionally or alternatively beprovided on the first surface 151 or the second surface 151. Therelative positions between the nose pad 141 and the spectacle lens 150may be fixed. That is, the relative positions between the reflectionunit 130 and the spectacle lens 150 may be fixed.

The spectacle lenses 150 may be held by the holding part 140. The anglebetween the holding part 140 and the spectacle lens 150 may beadjustable. The projection unit 120 may be held by the holding part 140.The position and attitude of the projection unit 120 may be adjustable.

In use, the nose pad 141 may be placed on the user's nose 165 and thetemple 143 on the user's ear 164. Relative positions of the holding part140, the spectacle lens 150, and the reflection unit 130 may be decidedbased on the position of the nose 165 and the ear 164 of the user 160.Relative positions of the holding portion 140 and the reflection unit130 may be substantially fixed.

Blurring caused by the image display apparatus 100 will be described.

An observation image 170 may include blurring, color shift, anddistortion because of an aberration caused by the lens system 123 and acomponent in the reflection unit, such as a lens or a half mirror.

The light including image information from the display 122 forms a groupof light rays from each pixel in the display 122. A light ray from apixel spreads as the ray travels. For example, the light ray spreads outin cone shape. A part of a light ray passes the lens system 123 and isreflected by the reflection unit 130 to be incident on the eye of a userand is collected on one point on the retina 163. In this case, it isable to observe the observation image 170 without blurring.

However, a part of a light ray may diverge from an ideal path because ofan aberration. In this case, light rays from a plurality of pixels mayoverlap each other on the retina 163, so that the user observes theobservation image with blurring.

To suppress such blurring, an image processing technique in which aninverse filter that has an inverse characteristic relative to a spacefilter or frequency filter that expresses the deterioration process thatcauses blurring in the observation image may be applied to the displayimage data. However, such processing with the inverse filter may resultin a pixel value that is out of the acceptable range for the image. Inparticular, an edge where a pixel value is near the border of theacceptable range is likely to include blurring. The acceptable range maybe set for each image or each image data. Circuit 110 may set theacceptable range for the input image data, the deblurring image data,and the display image data. The acceptable range may be stored in animage processing unit described later. The acceptable range may bedetermined prior to processing by the image processing unit.

A so-called Wiener filter may be applied to input image data of aprojector preliminarily to correct blurring of a focus on a screen. Inthe case where the pixel value after application of the Wiener filter isout of the acceptable range, (a) a clipping process may be executed forthe pixel value being out of the acceptable range or (b) contrast of theimage may be modulated so that the pixel values are in the acceptablerange. In the case of (a), the effect of deblurring may be suppressed,thus causing distortion of the image. In the case of (b), the blurringmay be corrected, however visibility of the observation image may below, due to a decrease in a contrast of the observation image and acontrast and luminance of the background. Contrast and luminance of thebackground refers to a maximum luminance of the observation image 160relative to the luminance of background. The higher the contrast andluminance of the background, the higher the visibility of theobservation image 170.

For this reason, it is desirable to display the observation imagewithout the blurring and other problems described above.

FIG. 2 is a block diagram of the image display apparatus according tothe first embodiment. FIG. 3 is a block diagram of a deblurring unitaccording to the first embodiment.

As shown in FIG. 2, the circuit (image processing device) 110 comprisesa background luminance input unit 210, an image processing unit 220, anda luminance correction unit (luminance setting unit) 230. For example,the background luminance input unit 210, the image processing unit 220,and the luminance correction unit (luminance setting unit) 230 may beincluded in the circuit 110.

The background luminance input unit 210 stores information (backgroundinformation) regarding a luminance of a background on which theobservation image 170 is overlapped and transmits the backgroundinformation to the image processing unit 220. The background luminanceinput unit 210, for example, may acquire the background information.

The image processing unit 220 comprises a deblurring unit 221 and acontrast conversion unit 222. The deblurring unit 221 corrects theblurring in the input image data that is input to the circuit 110 andtransmits the correction image data to the contrast conversion unit 222.The input image data and the correction image data include pixel values.The contrast conversion unit 222 modulates a contrast of the pixelvalues in the correction image data so that the pixel values are in theacceptable range and transmits the modulated correction image data as adisplay image data to the display 122 of the projection unit 120. Thecontrast conversion unit 222 also transmits information used as contrastconversion information to the luminance correction unit 230.

The luminance correction unit 230 transmits the luminance settinginformation to set the luminance of the projection unit 120 based on thecontrast conversion information. In particular, the luminance correctionunit 230 calculates luminance of the observation image 170, as decreasedby the modulating of the contrast of the pixel values in the correctionimage data, and transmits the result to the light source 121 of theprojection unit 120 as the luminance correction information.

The display 122 emits the light including image information to the lenssystem 123 by displaying the display image based on the display imagedata transmitted by the image processing unit 220, and transmittinglight from the light source 121 based on the luminance correctioninformation. The lens system 123 changes a traveling direction of atleast a part of the light including image information toward thereflection unit 130.

As described in FIG. 3, the deblurring unit 221 comprises filter imagefiltering units 330, 340. The deblurring unit 221 may further comprise afilter selection unit 310 and a filter switching unit 320. Each of thefilter image filtering units 330, 340 applies a different filter to theinput image data and transmits the correction image data. That is,deblurring unit 221 comprises a filter set consisting of two filters.The filter selection unit 310 transmits filter information to select afilter image processing unit based on the background luminanceinformation to the filter switching unit 320. The filter switching unit320 switches filters (the filter image processing units) based on thefilter information.

In this embodiment, a case in which a filter is selected from a filterset consist of two filters (filter image processing units) is described.A filter may be selected from a filter set consisting of three or morefilters.

The block diagrams in FIGS. 2 and 3 are examples only, and it is to beunderstood that the disclosed apparatus may be implemented in otherways. For example, a part of each block may be implemented apart fromthe image display apparatus.

FIG. 4 is a flow chart corresponding to the image display apparatus 100in FIG. 2. FIG. 5 is a flow chart corresponding to the FIG. 3.Processing in each of the blocks of the image display apparatus 100 isdescribed with reference to FIGS. 2 through 5.

In step S410, background luminance on which the observation image 170 isoverlapped (background luminance in front of the user 160) is input tothe background luminance input unit 210 and the background luminanceinformation is transmitted to the image processing unit 220. Thebackground luminance may, for example, be measured by a sensor such as acamera.

In the case where the environment of use changes momentarily, thebackground luminance may be measured every time the background luminancechanges and the measurement result may be input to the backgroundluminance input unit 210. In the case where the change of the backgroundluminance is small because of the limitation of the environment of use,the background luminance may be preliminarily measured and input to thebackground luminance input unit 210. In the case where the environmentof use is limited to a few patterns, the background luminance of eachpattern may be preliminarily measured and input to the backgroundluminance input unit 210.

The background luminance may be input manually or automatically. Forexample, a user 160 may input the background luminance to the backgroundluminance input unit 210, which may be implemented with software orhardware.

In step S420, blurring in the input image data is corrected based on thebackground luminance by the deblurring unit 221 and correction imagedata is transmitted to the contrast conversion unit 222. As describedabove, an image observed by a user may include blurring, color shift,and distortion because of an aberration caused by the lens system 123and reflection unit 130. To correct a blurring in the observation image,for example, the inverse filter that has inverse characteristic relativeto space filter or frequency filter that express the deteriorationprocess that causes blurring in the observation image may be applied tothe input image data. An energy function to express a difference betweena desirable image to be observed by a user, which does not includeblurring and the observation image 170 is made smaller by the inversefilter. Therefore, it is able to display a blurring-correctedobservation image to a user.

As described above, a pixel value of the correction image data may beout of the acceptable range. The more the inverse filter is effective tocorrect blurring, the more easily a pixel value of the correction imagedata may be out of the acceptable range because of the occurrence of anovershoot or an undershoot. In this case, if a contrast of thecorrection image data is modulated so that a pixel value is in theacceptable range, there is a problem that a contrast of the observationimage is decreased. The higher the background luminescence is, theharder it is for a contrast of the observation image to be affected bythe contrast conversion of the correction image data. For example, inthe case where the background luminescence is high and a contrast of thecorrection image data is modulated to decrease the background luminance,the decrease in a contrast of the observation image may be suppressed.For example, in the case where the background luminescence is high, aninverse filter that has strong effect for correcting blurring may beapplied.

An example of a way to calculate the inverse filter will be described.The deterioration process that causes blurring in the observation image170 may be expressed as a space filter. For example, a spot diagram maybe obtained by tracing a part of the light rays that are emitted from apixel in the display 122 and incident on pupils 162, by plotting pointsat the intersection of lines extended along the light rays from thereflection unit 130 to the pupils 162 in a reverse direction to thetraveling direction with the observation image 170. The spot diagram isa space filter expressed based on the deterioration process. A spacefilter may be obtained by imaging the observation image 170 with acamera and estimating the deterioration process.

The deterioration process may be expressed as below.y=Bx  (1)where y is a vector that expresses the observation image 170; x is avector that expresses a desirable image to be observed by a user thatdoes not include blurring; and B is a matrix of a space filter thatexpress the deterioration processes.

The inverse filter may, for example, be acquired by finding a correctionimage data to decrease the energy function J.J=∥x−B{circumflex over (x)}∥ ² +ε∥C{circumflex over (x)}∥ ²  (2)where {circumflex over (x)} is a vector that expresses the correctionimage data; C is a matrix that expresses regularization operator; and εis a regularization weight coefficient.

Matrix C may, for example, be a Laplacian operator or operators. Thefirst element in the relation (2) expresses energy of a differencebetween a desirable image to be observed by a user that does not includeblurring and the observation image 170. Because it is difficult for thecorrection image data to minimize the first item in an undeterminedsystem, the second element in relation (2) is a regularization item forfinding the correction image data stably. In this second element, thesmaller the regularization weight coefficient ε, the stronger thecorrection of the blurring in the observation image 170, but also thebigger the overshoot or undershoot around the edge of the correctionimage data. The overshoot or undershoot may cause a pixel value in thecorrection image data to fall out of the acceptable range. Theregularization weight coefficient ε may be, for example, selected basedon an edge intensity of the image. The weaker the edge intensity is, thesmaller the regularization weight coefficient ε may be.

The inverse filter is, for example, smaller than an energy function thatexpresses a difference between an image data that is generated byconvolution of the correction image data with the space filter thatexpresses the deterioration process that causes blurring in theobservation image and determined image data.

The inverse filter may be calculated by finding the correction imagedata to minimize the energy function J of the relation (2).I=B ^(T) B+εC ^(T) C)⁻¹ B ^(T)  (3)where I is a matrix that expresses the inverse filter.

In the case where the deterioration process attenuates high frequencycomponents of the image to be observed by an user, the inverse filtermay be, for example, a high-pass filter or a sharpening filter. Ahigh-pass filter or a sharpening filter may be implemented, for example,using an unshaped mask.

The debluring in step S420 will be described with reference to FIGS. 3and 5.

In step S510, the filter selection unit 310 transmits filter informationto the filter switching unit 320, so as to select a filter from a filterset consisting of two filters based on the background luminanceinformation. The filter set may, for example, comprise an inverse filterA that has strong deblurring effect (that is, having a small value of εin relation (2)) and an inverse filter B that has weak deblurring effect(having a large value of ε). For example, in the case where a backgroundluminescence is high, the inverse filter A (which has strong deblurringeffect) is selected and in the case where a background luminescence islow, the inverse filter B (which has weak deblurring effect) isselected. The filter set may, for example, comprise high-pass filtersthat have different pass levels or sharpening filters that havedifferent sharpening levels.

In step S520, the filter switching unit 320 switches filters to beapplied to the input image data based on the filter information.

In step S530, the filter selected by the filter switching unit 320 isapplied to the input image data. The way that the inverse filter isapplied to the input image data may depend on the type of the inversefilter, e.g., whether the inverse filter is a space filter or afrequency filter. For example, in the case where the filter set includesspace filters, the selected filter may be applied directly to the pixelin the input image data. In the case where the filter set includesfrequency filters, the selected filter may be, for example, applied tothe transformed input image data, which is transformed in a givenfrequency range with a Fourier transform. For example, in the case wherethe inverse filter is a space filter, the correction image data may begenerated by multiplying at least one of pixel values in the input imagedata and a weighting coefficient that expresses the inverse filter. Forexample, in the case where the inverse filter is a frequency filter, thecorrection image data may be generated by multiplying transformed imagedata obtained by Fourier transform of the input image data by theinverse filter, and by transforming the result of the multiplicationwith an inverse Fourier transform.

The inverse filter may be applied to a value in a color space that islinear with respect to the luminescence of the observation image 170.For example, a color space may be a CIE-RGB color system or a CIE-XYZcolor system. In the case where there is no linear relationship betweena pixel value of the input image data and a luminescence of theobservation image 170, the inverse filter may be applied to a valueobtained by transforming a pixel value of the input image data in acolor space that is linear to the luminescence of the observation image170. Then the value may be transformed in the original color space. Theinverse filter may be applied to a value in a color space that is notlinear to the luminescence of the observation image 170 as long as theeffect of the deblurring does not decrease undesirably.

In step 430, the contrast conversion unit 222 transmits the displayimage data, which is obtained by modulating the contrast of thecorrection image data so that a pixel value of the correction image datais in the acceptable range to the projection unit 120. The contrastconversion unit 222 also transmits the contrast conversion informationto the luminance correction unit 230.

The image processing unit 220 modulates a contrast based on the maximumvalue and the minimum value of the correction image data. For example,if the minimum value of the correction image data is out of theacceptable range, contrast conversion to the correction image data maybe performed according to relation (4).

$\begin{matrix}{{g\left( {x,y} \right)} - {\frac{MAX}{f_{\max} - f_{\min}}\left( {{f\left( {x,y} \right)} - f_{\min}} \right)}} & (4)\end{matrix}$The function f(x, y) is a pixel value of the coordinate (x, y) in thecorrection image data. The function g(x, y) is a pixel value of thecoordinate (x, y) in the display image data. The value f_(max) is themaximum value of the f(x, y). The value f_(min), is the minimum value ofthe f(x, y). The value MAX is the maximum value in the acceptable range.For example, an image has N-bit pixels, MAX is 2^(N)−1 because theacceptable range is from 0 to 2^(N)−1.

For example, in the case where the maximum pixel value of the correctionimage data is out of the acceptable range and the minimum pixel value ofthe correction image data is within the acceptable range, contrastconversion to the correction image data may be performed according torelation (5).

$\begin{matrix}{{g\left( {x,y} \right)} = {{\frac{{MAX} - f_{\min}}{f_{\max} - f_{\min}}\left( {{f\left( {x,y} \right)} - f_{\min}} \right)} + f_{\min}}} & (5)\end{matrix}$

For example, in the case where the maximum pixel value of the correctionimage data is in the acceptable range and the minimum pixel value of thecorrection image data is out of the acceptable range, contrastconversion to the correction image data may be performed according torelation (6).

$\begin{matrix}{{g\left( {x,y} \right)} - {\frac{f_{\max}}{f_{\max} - f_{\min}}\left( {{f\left( {x,y} \right)} - f_{\min}} \right)}} & (6)\end{matrix}$

Contrast conversion may be performed in other ways than the waysdescribed above as long as pixel values in the modulated correctionimage data are within the acceptable range. For example, contrastconversion may be performed based on the LUT (Look Up Table), whichassociates pixel values between pixel values before the modulation andpixel values after the modulation.

The pixel values of the correction image data may lie outside of theacceptable range as long as visibility of the observation image does notdecrease. The pixel values of the observation image that lie outside ofthe acceptable range may be clipped. For example, in the case where theminimum pixel value of the correction data is −100 and the maximum pixelvalue of the correction data is 500, contrast of the correction imagemay be modulated so that the minimum value is −5 and the maximum valueis 260. The pixel values that lie outside of the acceptable range may beclipped. If a number of the pixels whose pixel values are clipped issmall or a change of the pixel values by the clipping are small, themagnitude of the deblurring and the magnitude of the distortion byclipping are likewise small.

In the case where the pixel values of the correction data are in theacceptable range, contrast conversion may be performed with relation(4). In this case, visibility of the observation image will be improvedbecause of improvement of contrast of the observation image.

The contrast conversion information is information to indicate what kindof contrast conversion is performed. For example, in the case where thecontrast conversion is performed with relations (4) and (5), thecontrast conversion information may include the maximum pixel value ofthe correction image data. For example, the contrast conversioninformation may include information to indicate the amount of change ofthe maximum pixel value or the amount of change of the minimum pixelvalue of the correction image data.

In step S440, the luminance correction unit 230 finds an amount ofluminance correction to correct luminance of the light source in theprojection unit 120 so that the decrease of luminance of the observationimage by the contrast modulating will be canceled. The luminancecorrection unit 230 transmits the result as the luminance correctioninformation to the projection unit 120. For example, in the case wherethe contrast modulating is performed, e.g., according to relations (4)and (5), an amount of luminance correction may be found with relation(7).

$\begin{matrix}{{\Delta\; L} - {\left( {\frac{f_{\max}}{MAX} - 1} \right)L}} & (7)\end{matrix}$where L is a reference luminance value of the light source and ΔL is anamount of luminance correction.

A relationship between a pixel value of the correction data andluminance of light including image information from the display 122 maybe non-linear. For example, a relation between a pixel value of thecorrection data and luminance of light including image information fromthe display 122 may have gamma characteristic as below.

$\begin{matrix}{l = {L\left( \frac{f}{MAX} \right)}^{\gamma}} & (8)\end{matrix}$where l is a value of luminance of light including image information; fis a pixel value in the correction image; and γ is a parameter toexpress gamma characteristic. In this case, an amount of luminancecorrection may be calculated with the relation (9).

$\begin{matrix}{{\Delta\; L} = {\left( {\left( \frac{f_{\max}}{MAX} \right)^{\gamma} - 1} \right)L}} & (9)\end{matrix}$

In step S450, the light source emits light that is corrected based onthe luminance correction information toward the display 122. Forexample, in the case where the luminance correction information has anamount of luminance correction expressed with relation (7) or (9),luminance of light from the light source 121 may be corrected accordingto relation (10).L _(c) =ΔL+L  (10)where L_(c) is a luminance value of corrected light emitted from thelight source 121. The luminance of observation image 170 does notdecrease. Therefore, luminance and contrast of the background does notdecrease and contrast of the observation image 170 after the correctionincreases with respect to the contrast of the observation image 170before the correction. Therefore visibility of the observation image 170is improved.

In step S460, light including image information is emitted toward thelens system 123, when display image data input by the image processingunit 220 is displayed on the display 122. Light from the light source121, whose luminance is corrected based on the luminance correctioninformation, is transmitted to the display 122.

In step S480, the reflection unit 130 reflects at least a part of lightrays of the correction light toward the user's eyes 161. The correctionlight L3 is reflected by the reflection unit 130 so as to be incident onpupils 162 and is collected on the retina 163 by the lens and cornea sothat a virtual image is formed in front of the user 160. The reflectionunit 130 also transmits a part of the light incident on the reflectionunit 130. Therefore the user 160 is able to observe the observationimage 170 and outside world in front of the user.

By this embodiment, the user 160 is provided with an observation image170 in which the blurring due to aberration caused by such as the lenssystem 123 and the reflection unit 130 has been corrected. Furthermore,in the case where a pixel value after an inverse filter is applied isout of the acceptable range, the corrected observation image is providedwithout decreasing a luminance and contrast of the background. Thedecrease of the contrast of the observation image is further suppressed.

(Second Embodiment)

Principles of the distortion and color shift that may occur in the imagedisplay apparatus 100 will be described.

As described above, because of an aberration caused by transmission andreflection of light rays by the optical component and a lens system inthe reflection unit, deterioration such as blurring, color shift, anddistortion may occur. In this case, a user 160 observes the observationimage that includes more blurring, color shift, and distortion than maybe estimated to occur in the observation image.

The color shift and distortion in the observation image may be correctedby distorting the image inversely with respect to the color shift anddistortion caused by the aberration.

FIG. 6 is a block diagram of an example of the image display apparatusaccording to the second embodiment.

As illustrated in FIG. 6, the image processing unit 220 of the imagedisplay apparatus 100 further comprises a distortion correction unit710.

That is, the image processing unit 220 comprises the deblurring unit221, the distortion correction unit 710, and the contrast conversionunit 222. The deblurring unit 221 corrects blurring in the image datainput to the image display apparatus 100 and transmits the correctionimage data to the distortion correction unit 710. The distortioncorrection unit 710 corrects distortion in the correction image data andtransmits distortion correction image data to the contrast conversionunit 222. The contrast conversion unit 222 transmits display image datato the projection unit 120 and transmits contrast conversion informationto luminance correction unit 230.

The block diagram illustrated in FIG. 7 is exemplary and is not requiredfor implementation. For example, a part of each block may be separatedfrom the image display apparatus. The burring correction unit 221 andthe distortion correction unit 710 may be substituted for each other. Inthis case, the distortion correction unit may correct distortion in theinput image data and transmit distortion correction image data to thedeblurring unit 221. The deblurring unit 221 may correct blurring in thedistortion correction image data and may transmit deblurring image datato the contrast conversion unit 222. The contrast conversion unit 222may transmit display image data in which the contrast of the deblurringimage data is modulated to the projection unit 120.

The deblurring unit 221 and the distortion correction unit 710 may besubstituted by a unit that is able to correct both blurring anddistortion. In this case, the unit may correct blurring and distortionin the input image data and transmit correction image data to thecontrast conversion unit 222. The contrast conversion unit 222 maytransmit display image data in which the contrast of the correctionimage data is modulated to the projection unit 210.

FIG. 7 is a flow chart to illustrate the image display apparatus 100according to the second embodiment. Process in the image processing unit220 will be described with FIGS. 6 and 7.

In step 810, the deblurring unit 221 corrects blurring in the inputimage data input to the image display apparatus 100, based on thebackground luminance, and transmits the correction image data to thedistortion correction unit 710. The blurring may be corrected in thesame manner as described above with respect to the first embodiment.

In step S820, the distortion correcting unit 710 corrects distortion ofthe correction image data and transmits the distortion-corrected imagedata to the contrast conversion unit 222. For example, color shift anddistortion due to aberration are expressed by an LUT indicating arelation between arbitrary pixels in the display 122 and the positionsin the observation image corresponding to the pixels. The deblurringimage data is distorted inversely to a color shift and distortion basedon the LUT. Therefore, a user is able to observe the observation imagethat does not include color shift and distortion.

In step S830, the contrast conversion unit 222 transmits the displayimage data in which contrast is modulated so that pixel values of thedistortion correction image are modulated to be in the acceptable rangefor the projection unit 120. The contrast conversion unit 222 alsotransmits the contrast conversion information to the luminancecorrection unit 130. The contrast of the correction image data may bemodulated, for example, in the same manner as described above withrespect to the first embodiment.

By this embodiment, the user 160 p is provided with an observation image170 in which blurring due to aberration caused by color shift anddistortion of the lens system 123 and the reflection unit 130 have beencorrected. Furthermore, in the case in which a pixel value after aninverse filter is applied is out of the acceptable range, the correctedobservation image is provided without decreasing a luminance andcontrast of the background. The decrease of the contrast of theobservation image is also suppressed.

FIG. 8 is a diagram of an example of a structure of the image displayapparatus. As described in FIG. 8, the circuit 110 includes, forexample, an interface 610, a processing circuit 620, and a memory 630.

The circuit 110 may, for example, be connected to such as a recordingmedium that is outside of the image display apparatus, network, or animage reproducer, and acquire the input image data through an interface.The circuit 110 may be connected to a recording medium, network, or animage reproducer by wire or by a wireless network.

The processing circuit 620 may, for example, process the input imagedata information acquired through a sensor 650, and image informationbased on the program 640. For example, processes of the image processingunit 220 and luminance correction unit 230 may be performed in theprocessing circuit 620 based on the program 640.

For example, a program 640 to process acquired input image data may bestored in the memory 630. The memory 630 may, for example, comprise arecording medium recordable magnetically or optically and be able torecord the acquired input image data, information acquired through asensor 650, and image information. For example, the memory 630 mayrecord a program 640 to control the image display apparatus 100 orvarious setting information. The input image data may be transformedbased on the program 640 and display image data may be generated. Imageinformation such as the input image data and display image data may bestored in the memory 630. The program 640 may be preliminarily stored inthe memory 630. The program 640 may be provided through a memory mediumsuch as CD-ROM (Compact Disc Read Only Memory) or network and installedin the memory 630.

The circuit 110 may include a sensor 650. The sensor 60 may be anarbitrary sensor such as a spectral radiance meter, a spectroradiometer,a camera, a microphone, a position sensor, or an acceleration sensor.For example, background luminance in front of the user may be measured.For example, an image to be displayed on the display 122 may be changedbased on the information acquired by the sensor 650. Convenience andvisibility of the image display apparatus may be improved.

An integrated circuit (IC) or IC chip set such as LSI (Large ScaleIntegration) may be used as a part or whole of each block in the circuit110. Each block may be implemented by an individual circuit. A part orwhole of each block may be implemented by an integrated circuit. Blocksmay be implemented as a unit. A block or certain blocks may be separatedfrom the other blocks. A part of a block may be separated from the otherpart of the block. A dedicated circuit or general processor may be usedin the circuit 110, rather than LSI circuitry.

In the image display apparatus described above, processing of thecircuit 110, the image processing unit 220, and luminance correctionunit 230 may be performed by a processor such as CPU in ageneral-purpose computer executing a program. The program may bepre-installed in the computer. The program may be provided through amemory medium such as CD-ROM or network and installed in the computer.

In the disclosed embodiments, all or part of the processing may beperformed by an OS (Operating System) operating in a computer based on ainstruction from the program installed in a computer, or by anintegrated system memory medium, a database management software,Middleware (MW) such as network, etc.

A memory medium is not limited to a medium independent from a computeror an integrated system; a memory medium memorized by downloading aprogram through LAN or internet may also be used. A plurality of memorymedia may be used for processing in these embodiments.

A computer or an integrated system may execute each disclosed processbased on a program memorized in memory media. A computer or anintegrated system may be implemented by an apparatus comprising apersonal computer or microcomputer, or a system comprising a pluralityof apparatus connected to a network. The computer may be implemented notonly by a personal computer, but also by a image processing unit ormicrocomputer included in an information processing equipment. Acomputer is any device that is able to perform the functions describedin these embodiments.

Each of the embodiments was described with specific examples. However,this disclosure is not limited to these specific examples. For example,one of ordinary skill in the art will understand that this disclosuremay be implemented using available variations in the specificcomposition of each element like the background luminance input unit,the deblurring unit, the contrast conversion unit, the luminancecorrection unit, the light source, the display, the lens system, and thereflection unit.

One of ordinary skill in the art will also understand that thisdisclosure may be implemented using combinations of two or more elementsfrom the specific examples.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the disclosure. Indeed, the embodiments described herein may beembodied in a variety of other forms; furthermore, various omissions,substitutions and changes in the form of the embodiments describedherein may be made without departing from the spirit of the disclosure.The accompanying claims and their equivalents are intended to cover suchforms or modifications as would fall within the scope and spirit of thedisclosure.

The invention claimed is:
 1. An image processing device comprising: abackground luminance input circuit configured to store backgroundluminance information regarding background luminance; an imageprocessing circuit configured to generate display image data andcontrast conversion information, the display image data being providedby modulating a contrast of a correction image data so that pixel valuesincluded in the correction image data are in an acceptable range, thecorrection image data being provided by correcting blurring in an inputimage data to be input to the image processing circuit based on thebackground luminance information, the contrast conversion informationbeing information used for the modulating, wherein the image processingcircuit comprises: at least two filter image processing circuitsconfigured to apply filters to be applied to the input image data, andtransmit the correction image data; and one of the filters is a filterto decrease an energy function that expresses a difference between animage data that is generated by convolution of the correction image datawith a space filter that expresses a deterioration process to causeblurring in an observation image and determined image data; and aluminance setting circuit configured to generate a luminance settinginformation to set a luminance of a projection unit based on themodulation information.
 2. The device according to claim 1, wherein theimage processing circuit further comprises: a filter selection circuitconfigured to transmit filter information to select one of the filterimage processing circuits; and a filter switching circuit configured toswitch the filter image processing circuits based on the filterinformation.
 3. The device according to the claim 2, wherein the filterimage processing circuit that has stronger correction effect is selectedby the filter selection circuit.
 4. The device according to the claim 1,wherein one of the filters is a filter to be applied in color spacelinear to luminance of the projection unit.
 5. The device according toclaim 1, wherein the contrast conversion information includes themaximum pixel value or the minimum pixel value in the correction imagedata.
 6. The device according to claim 1, wherein the contrastconversion information includes information to indicate an amount ofchange of the maximum pixel value of the correction image data by themodulation.
 7. The device according to claim 1, wherein the contrastconversion information includes information to indicate an amount ofchange of the minimum pixel value of the correction image data by themodulation.
 8. The device according to claim 1, wherein the luminancesetting information is generated based on a decrement of luminance dueto the modulation of contrast of the correction image data.
 9. Thedevice according to claim 1, wherein the image processing circuit isconfigured to modulate contrast of the correction image data afterdeburring in the correction image data.
 10. An image display apparatuscomprising: an image processing device comprising: a backgroundluminance input circuit configured to store background luminanceinformation regarding background luminance; an image processing circuitconfigured to generate display image data and contrast conversioninformation, the display image data being provided by modulating acontrast of a correction image data so that pixel values included in thecorrection image data are in an acceptable range, the correction imagedata being provided by correcting blurring in an input image data to beinput to the image processing circuit based on the background luminanceinformation, the contrast conversion information being information usedfor the modulating, wherein the image processing circuit comprises: atleast two filter image processing circuits configured to apply filtersto be applied to the input image data, and transmit the correction imagedata; and one of the filters is a filter to decrease an energy functionthat expresses a difference between an image data that is generated byconvolution of the correction image data with a space filter thatexpresses a deterioration process to cause blurring in an observationimage and determined image data; and a luminance setting circuitconfigured to generate a luminance setting information to set aluminance of a projection unit based on the modulation information; theprojection unit to emit light including image information based on a thedisplay setting information; and a reflection unit to reflect at least apart of light emitted by the projection unit toward eye and transmit apart of incident light.
 11. The apparatus according to claim 10, whereinthe projection unit comprises: a light source configured to emit lightbased on the luminance setting information; and a display to display animage based on the display image data.
 12. The apparatus according toclaim 10, further comprising a sensor to detect background luminance,wherein the sensor inputs the background luminance to the backgroundluminance input circuit.